Tag: large-batch training

Optimizing MLPerf Training closed-division submissions: what the literature shows

Which methods documented in the literature most reduce time-to-quality for MLPerf Training closed-division submissions, given the rule that model, optimizer and most hyperparameters are fixed?

A review of 47 sources on how to minimize time-to-quality in the MLPerf Training closed division, where the model, optimizer, and most hyperparameters are fixed by the rules. The levers the evidence supports are: picking the global batch size that sits at the reference-convergence boundary, reduced-precision training (bf16/fp8, now also fp4), distributed-parallelism and communication-overlap engineering, data-pipeline and kernel work, and benchmark-specific plays for DLRM, LoRA fine-tuning, and MoE pretraining. Each lever is worth roughly 1.2–2× in isolation and they compound, but the convergence-speed cap (Reference Convergence Points) means the winning move is maximum throughput at the RCP boundary, not faster convergence. Confidence is moderate: the per-lever results are replicated systems literature, but no public study directly tests the full closed-division strategy.

MLPerf · distributed training · mixed precision · large-batch training · communication overlap · HPC